The present invention relates to computer system testing, and more particularly, this invention relates to command line interface permutations.
Command line interfaces (CLIs) typically have many parameters per command to choose from, with each of those parameters having many values that the parameter can accept and possibly an infinite number of invalid values. Moreover, CLIs usually have a big command set, with one command for each important function the CLI can perform. Providing manual input for all the commands, parameters for each command, and values for each parameter to drive testing of the CLI under test conditions can be a daunting, if not improbable task. Typically, there are five reasons why the task is so difficult:                1) Manually typing the commands opens up the possibility for the tester to mistype any part of the command string, such as the command itself, parameters, values, etc., and this can result in a needless expenditure of valuable time, as the command will not run properly and/or as anticipated.        2) If the commands are entered manually, then all the time that is used to enter the command strings is generally excessive and wasted, since a human is slower (by several orders of magnitude) than a modern processor that can input the commands in place of a human tester.        3) Unless an extremely effective and reliable process is used to track and organize which commands have been run, generally a subset of the commands will have to be rerun, wasting additional time. This is because it is very difficult for a human to keep track of which commands have been run, even with the aid of tools like spreadsheets, since by the very nature of the problem, the quantity of commands grows exponentially with the number of values per parameter, parameter per command, and commands that the CLI supports.        4) Manually analyzing the result of the command entered and verifying whether it succeeded or failed is almost always repetitive and can be better handled by automation inline with the command's execution.        5) During development of the CLI and other components of the system that the CLI interacts with, executing some commands may make the CLI hang (stop functioning properly) and the test must be re-started from the beginning, losing all work completed up to that point. Automating this “cleanup-and-install” task in addition to logging the failed execution and skipping to the next execution is a repetitive task that can take additional time to do.        
Therefore, a solution that can address the shortcomings of current methodologies and systems to test commands would save enormous amounts of time and energy on the part of human testers.